feat(tracing): Enable Sentry span streaming mode - #8245
Merged
Conversation
Turn on `trace_lifecycle="stream"` so spans are sent to Sentry in batches as they finish, instead of buffering a whole trace in memory and shipping it as a single transaction event. This removes the 1000-span-per-transaction cap, keeps memory flat in long-running consumers, and preserves spans that already finished when a process is killed mid-trace (e.g. an OOM-kill). Stream mode disables the legacy tracing API, and it fails silently: `start_span()`, `start_transaction()` and `update_current_span()` become no-ops, and `get_current_span()` / `scope.span` / `scope.transaction` return None. Every call site therefore has to move to `sentry_sdk.traces` in the same change or tracing goes dark with no error. `op` becomes the `sentry.op` attribute, `description` becomes the span name, and `set_data`/`set_tag` on a span become `set_attribute`. Scope tags and scope attributes reach disjoint destinations: tags land on error events only, attributes on spans only. Query-path tags are converted to attributes so they stay queryable on trace data; the consumer and error-reporting paths keep `set_tag`, since they have no active span and would otherwise lose the data entirely. Three things beyond the mechanical translation: - Several spans passed raw SQL as `description`, which becomes the span name -- the grouping key -- under stream mode. Give them stable names and move the SQL to a `db.query.text` attribute. - `metrics.incr`/`metrics.timing` no longer exist in sentry-sdk 2.66.1, so SentryMetricsBackend was a latent AttributeError. Map them to `count` and `distribution`. - Fix three pre-existing Mapping-vs-dict type errors in ClickhouseConnectPool. They were previously unreported because the mypy pre-commit hook only checks changed files, and this change touches that file. Drop the on-demand profiler: it drives `Transaction._profile`, which has no stream-mode equivalent. Note that `SENTRY_TRACE_SAMPLE_RATE` is 0 in every checked-in settings file, so this emits nothing until a deployment sets it. The CLI-init transaction previously forced `sampled=True`; stream mode has no equivalent, so it now obeys the sample rate like every other span. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
pbhandari
reviewed
Jul 30, 2026
alexander-alderman-webb
left a comment
alexander-alderman-webb
left a comment
There was a problem hiding this comment.
Looks good to me overall.
The two bot comments look valid.
The main possibly disruptive change is calling Scope.set_attribute() where there was previously Scope.set_tag(), which includes edits of sentry_sdk.set_tag() -> sentry_sdk.set_attribute().
Keep in mind that attributes on the scope are applied to logs, metrics, and (streamed) spans, whereas tags are applied to exceptions. Completely replacing set_tag() with set_attribute() results in the data not showing up on exceptions while the scope is active.
Dual-write scope tags and attributes on API error-enrichment paths so values still land on Sentry issues during the tags→attributes transition. Centralize the stream-mode op key as SENTRY_OP, harden querylog trace_id lookup, clarify clickhouse-connect param narrowing, and always set insert query_id on spans.
client.insert takes a row matrix rather than SQL, but it still builds and sends "INSERT INTO <table> (<cols>) FORMAT Native" on the wire, so the statement was available all along. Reconstruct it for db.query.text instead of leaving the attribute unset. This also brings the two pools in line: ClickhousePool.insert routes through execute() with an explicit statement and already populated the attribute, so only the HTTP path was missing it. Row data stays out of the attribute since it is unbounded and may hold PII.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 2087ec8. Configure here.
Bugbot caught five scope-level set_tag call sites the sweep converted to set_attribute only: slo_status, cache_status, query_size_group, and the querylog duration_group/timeout/experiments/max_threads block. Scope tags and attributes reach disjoint destinations, so attribute-only means these stop appearing on Sentry issues. The API error-enrichment paths already dual-write via set_tag_and_attribute; these were missed. Route them through the same helper so no bare sentry_sdk.set_attribute is left outside it. Verified against a capturing transport that the value lands on both the error event tags and the streamed span attributes. Also normalize the referrer once in _trace_transaction so the transaction name matches the referrer attribute instead of interpolating a literal "None" when the Referer header is absent.
The SDK annotates metrics attributes as an invariant dict while Tags is a Mapping, so the call needed narrowing. _attributes did that with dict(tags), which is a redundant copy: _capture_metric already iterates .items() into a fresh dict and never retains or mutates the caller's mapping. Cast instead. Verified the SDK accepts a non-dict Mapping at runtime and that all four metric types still emit with correct type, unit, and attributes. Narrowing the override signatures instead would violate LSP against MetricsBackend, and callers reach this through the abstract interface, so the conversion has to stay at the call boundary.
pbhandari
approved these changes
Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Snuba runs the Sentry Python SDK in transaction mode: spans accumulate in memory and ship as one transaction event when the root span ends. Stream mode (
trace_lifecycle="stream") sends them in batches as they finish, which gets us:Why it's one big change
Stream mode disables the legacy tracing API, and it fails silently.
start_span(),start_transaction()andupdate_current_span()become no-ops;get_current_span(),scope.spanandscope.transactionreturnNone. No exception, no warning at runtime. So every call site has to move tosentry_sdk.tracesin the same commit or tracing just goes dark.The translation:
op→sentry.opattribute,description→ spanname,set_data/set_tagon a span →set_attribute.StreamedSpangenuinely has noop,description,set_data,set_tag, orstart_child, so mypy catches most of the sweep.Tags vs. attributes
Worth a close look in review. Verified empirically against 2.66.1: scope tags and scope attributes reach disjoint destinations — tags land on error events only, attributes on spans only. Since stream mode emits no transaction event, query-path tags had to become attributes to stay queryable on trace data.
But the consumer and error-reporting paths (
cli/consumer.py,lw_deletions_consumer.py,consumers/consumer.py,clickhouse/http.py) have no active span at all — no WSGI middleware, no root segment. Converting those would have sent the data nowhere, so they keepset_tag, with a comment explaining why.Beyond the mechanical translation
description, which becomes the span name — the grouping key — under stream mode. Unbounded cardinality. Now stable names with the SQL indb.query.text, matching what the SDK itself does.AttributeError.metrics.incr/metrics.timingdon't exist in sentry-sdk 2.66.1, soSentryMetricsBackendwould raise on any deploy withDOGSTATSD_SOCKET_PATHset. Mapped tocount/distribution.ClickhouseConnectPool. Previously unreported because the mypy hook only checks changed files; touching that file surfaced them, so they're fixed here.Transaction._profile, which has no stream-mode equivalent. Removed the module, itsondemand_profiler_hostnamesoption, and the docs section.Verification
mypy .— clean across all 1079 files (was 3 errors on master).-p no:warningsinpyproject.tomlmakes-W error::...inert, so this needed anaddoptsoverride — confirmed with a deliberate canary that the guard actually fires before trusting the green.http.serversegment with the migrated attributes; child spans nest correctly.trace_idverified at sample rate 0 (the default) —NoOpStreamedSpaninherits the propagation context's id, so it stays a real 32-hex value. The RPC path now reads it from the propagation context directly, which would otherwise have silently started writing"".Deploy note
SENTRY_TRACE_SAMPLE_RATEis 0 in every checked-in settings file, so this emits nothing until a deployment sets it (configured in ops, not here). One behavior change to be aware of: the CLI-init transaction usedsampled=Trueto bypass sampling, and stream mode has no equivalent — it now obeys the sample rate like every other span. Thesnuba_init_timemetric is unaffected.🤖 Generated with Claude Code